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Executive Summary 


As the largest higher education institution in the golden state, the California State 
University (CSU) System has been championing advancements in instructional technology 
and teaching innovations across face-to-face, interactive television, hybrid, and online 
platforms. This report focuses on needs pertinent to the southern San Joaquin Valley and 
Antelope Valley. Thus, the report begins with an Introduction section describing the 
service region of CSU, Bakersfield (CSUB) in support of the Quality Assurance (QA) 
training for instructors and course certification for the institution. Recent CSUB initiatives 
are addressed in the Campus QA Background section to examine indicators of the ongoing 
capacity building in Academic Year (AY) 2016-17. In addition, QA Professional 
Development results are articulated with an overview of course certification status to 
facilitate analyses of student learning outcomes. Furthermore, contextual information is 
incorporated from the institutional research office to assess the impact of online and hybrid 
course offerings on students from diversified demographic backgrounds. The university 
transcript records are extracted to confirm the comparison of student performance in online, 
hybrid, and other course settings. Survey data are analyzed from 103 students who enrolled 
in online and/or hybrid courses taught by 10 instructors in AY 2016-17. The report 
concludes with recommendations for sustaining ongoing progress of the SQuAIR project 


at CSUB and beyond. 
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Introduction 

The Student Quality Assurance Impact Research (SQuAIR) project is led by the 
Chancellor’s Office of the California State University (CSU) to determine the impact of 
Quality Assurance (QA) professional development and course certification on teaching 
performance and student success. The project goals are aligned with the university 
commitment to maximizing student success, reducing time to degree, improving 
graduation rates, and shrinking the achievement gap. 

California State University, Bakersfield (CSUB) participated in the SQuAIR 
partnership building with four sister campuses (CSU Channel Islands, East Bay, Fullerton 
and San Francisco) since January 2017 to improve its QA services through online and/or 
hybrid instructional practices. The support for teaching and professional development has 
created a community of faculty learners to improve student academic performance and 
course completion rates. 

As the only public university within a radius of two-hour driving in all directions, 
CSUB has as its primary responsibility to serve the communities throughout the southern 
part of the California Central Valley and the Antelope Valley. Online instruction is an 
important platform for CSUB to offer higher education opportunities for over 1.3 million 
local residents across a land area as large as the state of Massachusetts. As Chancellor 
White (2016) acknowledged, “Cal State Bakersfield has lifted up lives, families and 


communities... and is a catalyst for innovation, social justice and economic success”!. 


1 https://www2.calstate.edu/csu-system/chancellor/the-chancellors-communications/Pages/csub-presidents- 
associates-dinner.aspx 


In support of the program outreach, CSUB volunteered to be an original partner with 
California State University, Northridge to initiate the CSU system-wide partnership 
building for online teaching. Prior to its incorporation of learning management systems, 
such as WebCT and Blackboard, the campus already invested in its infrastructure building 
to support course offerings through Instructional Television Network (ITV) at its 
surrounding community colleges. While technology advancement continues on the ITV 
platform to feature interactive two-way video instruction’, the campus has institutionalized 
professional training for online and hybrid instruction, and thus, add new venues to support 
student program completion. According to the unduplicated counts of CSUB enrollment 
in AY 2016-17, online and hybrid instructions have grown to serve over 71% of the 
students beside the face-to-face teaching sector. In particular, Figure 1 showed that the 
head count has surpassed the ITV enrollment nearly 7.5 times, which directly supported an 


emphasis of the SQuAIR student survey on online and hybrid instruction in Summer, 2017. 
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Figure 1. Distribution of Student Enrollment in Non-Face-to-Face Settings 


? https://www.csub.edu/its/services/itv/index.html 


In comparison, the online enrollment figure was more than twice of the hybrid 
enrollment counts (Figure 1). Altogether a total of 65 courses/177 course sections were 
offered online in Fall, 2016 and Spring, 2017 while 54 courses/114 course sections were 
offered hybrid in Fall, 2016 and Spring, 2017. Primary reasons for the growth of online 
enrollment hinged on the QA background and QA professional development at CSUB. 

Campus QA Background 

As online education evolved into a popular mode of instruction in higher education, 
investing in quality assurance has become critical in meeting the increasing demands for 
high quality instruction. In fulfilling its commitment to professional practice, CSUB 
became a Quality Matters (QM) subscriber in May of 2012. In the spring of 2013 the 
Academic Senate approved an Online and Hybrid Instruction Policy that required all 
faculty to receive training prior to teaching online or hybrid courses. The Distributed 
Learning Committee appointed by President Mitchel proposed a set of requirements for 
online and/or hybrid instruction, including demonstration of competencies in using 
Learning Management Systems. 

To facilitate faculty preparation, professional development programs are designed 
and implemented to provide the much-needed training for course preparation and QM 
review by an internal or external team. The internal review is conducted by experienced 
faculty whose course(s) were recognized through external QM review. The campus also 
contributes $1000 to pay the process fee for external QM certification. Training 
workshops, Applying the QM Rubric (APPQMR), are developed by local experts and 


offered regularly in each term to guide the instructional design of quality online courses. 


Additional training is provided through delivery of Peer Reviewer Course (PRC) and QM 
Master Reviewer Course (QMMRC) to prepare QM certified reviewers and master 
reviewers. CSUB also actively participated in the CSU system Quality Online Learning 
and Teaching (QOLT) initiative. As of October 10, 2017, credentials of the online- 


teaching faculty have been achieved across all of the following professional domains (see 


Table 1). 
QA Faculty Resources Number 

APPQMR Completer 252 

QOLT Completer 6 

QM Certified Reviewer 10 

QM Master Reviewer 3 
Internal Course Review Completer 212 

QM Certified Courses 15 

QM Training Facilitator i 


Table 1: QA Capacity of CSUB Faculty in Fall, 2017 


QA Professional Development at CSUB 
Currently, three hundred twenty faculty and staff have received various QM 
training; fifteen courses received QM certification; and two hundred twelve courses were 
approved through internal reviews process. The QA professional development directly 
strengthened faculty expertise in their course offerings. In AY 2016-17, CSUB offered 
795 course sections, and 80.25% of them were taught by instructors with QM training. One 


hundred and thirty-nine instructors taught at least one online or hybrid course. One 
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hundred and six of them received QM training and 19 instructors had at least one course 
approved by the internal review through independent study of the QM rubric. 

The Summer Institute for Online Teaching is viewed as a flagship model for 
campus-wide training and faculty professional development. The number of participants 
consistently increases each year. All four academic schools are well represented in the 
Institute. Feedback received from attendees has been overwhelmingly positive. The goal 
of the Summer Institute, administered by the Faculty Teaching and Learning Center 
(FTLC), is to align the activities of the Institute with the University’s strategic goals. The 
training is balanced to support both online and hybrid instructions. For instance, among 
the 139 instructors who taught online and/or hybrid courses in AR 2016-17, 68 taught 
hybrid course sections only, 61 taught online course sections only, and 10 taught both 
online and hybrid sections. Because some courses were offered in multiple sections, a total 
of 56 hybrid courses and 59 online courses were taught by instructors with CSUB QM 
training, which concurred with the balanced support of faculty professional development 


in the course section counts. 


Impact of Online and Hybrid Courses on Student Learning Outcomes 

In AY 2016-17, CSUB offered a total of 978 courses face-to-face, 40 courses 
through ITV, 81 courses hybrid, and 92 courses online. The percent of students receiving 
a lettered grade across these teaching platforms are listed in Table 2. 

Regardless of the teaching platforms, noted in each column of Table 2, student 


consistently earned grades in the A grade category. In comparison, the percent of students 
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achieving an A in a face-to-face class is slightly lower than their peers in an online or hybrid 
class. Meanwhile, a higher percent of students received an Incomplete (1) or Report in 
Progress (RP) grade in a hybrid class than in other teaching platforms. Online and face- 
to-face classes seem to have a larger chance to grant a No Credit (NC), Withdrawal (W), 
or Unauthorized Withdrawal (WU) grade (Table 2). 


GRADE FACE-TO-FACE ITV HYBRID ONLINE 


A 20.86 22 Se, 2229) 
A- 9.42 18.66 12.47 11.30 
B+ 7.54 ee 8.76 OS 
B 12202 11.98 10.66 10.99 
B- 6.96 8.67 TT. 8.08 
C+ 5.03 5.48 5.00 5.15 
C 8.34 7.10 4.60 5.43 
C- 3.89 4.33 2.87 2.61 
D+ 1.68 0.90 1.09 2? 
D S22, 1.26 1.95 1.86 
D- 1.28 1.26 1.38 0.80 
F (fy 2D, 4.42 6.26 
I 0.42 0.66 howe) 0.26 
NC 1.26 0.12 0.52 2idD 
RD 0.04 0.00 0.00 0.00 
RP 0.01 0.00 0.43 0.00 
Ww IQ? 0.48 0.55 0.89 
AU 0.03 0.00 0.00 0.00 
WU 1.66 Is? 1.18 1.98 


Table 2. Percent of the lettered grade from face-to-face, ITV, hybrid, and online courses 


In Table 3, results are aggregated to compare the percent of students receiving a 
passing grade of C or above across the entire CSUB student population. The results show 
a higher percent of students passing a hybrid or online class than a face-to-face class. 

GRADES FACE-TO-FACE ITV HYBRID ONLINE 


C OR ABOVE | Teal 84.29 79,32 72.96 
Table 3. Percent of student receiving a passing grade or above across teaching platforms 
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In summary, online and hybrid instruction, as an alternative to face-to-face teaching, 
not only surpassed the enrollment of ITV that was established in the mid-1990s, but also 
provided a better chance than the face-to-face alternative to support student course 
completion at CSUB (Table 3). 

Comparison of GPA Outcomes across Student Demographics 

In the CSUB service region, Kern County has been ranked as one of the lowest 
regions in access to adult education across the United States (Brookings Institution, 2010), 
and Bakersfield was ranked as one of the least educated metropolitan areas in the nation 
(Zumbrun, 2008). The course enrollment data in AR 2016-17 also showed that 63.6% of 
the CSUB student population belong to underrepresented minority (URM) groups. In 
addition, 61.6% of the local enrollments were females, 57.4% were Pell-eligible, 56.8% 
had Hispanic/Latino origin, and 30.3% of the student population experienced remediation 
training in English and/or mathematics. With the grade points configured for the letter 
grade scale, the average grade point (GPA) is presented for these demographic 


categorizations across different teaching platforms (Table 4). 


Demographics Face-to-Face ITV Hybrid Online 
Minority URM 20, 2.92 2.88 2.76 
Representation Non-URM 2.89 3.14 3.27 2.96 
Female 2.74 3.01 3.07 2.88 
Gender Male 2.58 DOF 29) oe fe) 
Yes 2.54 291 2072 213 
Pell-Eligible No 2.87 eh eo 3.00 
English and Math pia ls) 2.61 2.28 2.46 
Remediation English Only 2.41 2.72 249 Pe 
Math Only 223 29h -. 250 2.47 


None 2.87 3.07 3.25 2.93 
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American Indian 2.81 Sl — 332 249 

Asian 2.83 2.99 3.06 2:97 

African American 2.38 212 241 2.40 

Ethnicity Hispanic/Latino 2d D5.” 2,90 Doe 
Native Hawaiian 2077 3.30 863.68 2.93 

White 2.96 3.19 3.41 3.01 

Multiple Races 2.81 3.10 3.03 2.91 


Table 4. GPA distributions across instructional mode and demographics. 

The choice of demographic variables in Table 3 was guided by a document of the 
CSU Chancellor’s Office, Outline for Reporting SQuAIR Results, August 2017. In 
reviewing of that document, the CSUB research team attempted to obtain student data on 
first generation status. A colleague of the CSUB’s OIRPA office responded, 

Students are not required to provide first generation status data in their CSU 

Mentor/Cal State Apply application; hence, that is the reason why there are gaps for 

this particular variable. Without a response from a student, we have no way of 

knowing whether they are a first generation student. (Personal Communication on 

10/3/2017). 

To amend the data gap, additional information gathering would demand extra time 
to establish a protocol for IRB approval, which will far exceed the one-week workload for 
the report completion. Given the time constraint, interpretation of the GPA comparisons 
is delimited to these demographic variables aggregated in Table 4: 

e On the URM dimension, independent sample t tests were conducted in each teaching 
platforms to show a significant gap of GPA between URM and Non-URM students 


at a=.0001. Despite the consistent difference, the results in Table 4 demonstrated 


14 


higher GPAs for online and/or hybrid students than the GPA for their peers in face- 
to-face courses. 

e Regarding the gender difference, statistical testing indicated a significantly higher 
GPA for female students than male students across face-to-face, hybrid, and online 
platforms. Students in ITV course sections did not demonstrate significant gender 
differences in the GPA records. On average, students in online and/or hybrid 
courses had a higher GPA than students in face-to-face classes, regardless of the 
gender grouping. 

e The Pell-Eligible criterion showed a significantly lower GPA for the eligible group 
across all teaching platforms. In comparison, GPA for face-to-face students was 
lower than the other student groups. 

e A statistical F test indicated significantly higher GPAs for students without 
remediation requirements. Students with both remediation courses in English and 
mathematics had the lowest GPAs across the four teaching platforms in Table 4. 
Again, students in hybrid or online classes achieved higher GPAs than their peers 
in a face-to-face class. 

e On the ethnicity dimension, African American and Hispanic/Latino groups obtained 
lower GPAs than other groups in Table 4. Except for the category of American 
Indian, GPA of face-to-face students were lower than that of their peers in the 
corresponding ethnic groups under other teaching platforms. 

In summary, student learning outcomes, as represented by the GPA index, 


demonstrated better course performance for the online and/or hybrid groups than the face- 
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to-face group across the demographic classifications of student URM, gender, Pell- 
eligibility, and remediation status. 
SQuAIR Survey Results 

Five CSU campuses participated in collection of Student Course Survey Data 
(SCSD) to evaluate student learning experiences in both online and hybrid courses taught 
by faculty who have successfully completed Quality Assurance training and/or 
certification. The instrument is aligned with the QM CORE/QOLT essential standards, 
and has been administered in Summer, 2017. A total of 103 CSUB students responded to 
this survey. The 25 survey items are classified into eight groups to reflect student course 
experiences in (1) Course Overview and Introduction, (2) Assessment of Student Learning, 
(3) Instructional Materials and Resources Utilized, (4) Student Interaction and Community, 
(5) Facilitation and Instruction, (6) Technology for Teaching and Learning, (7) Student 
Support and Resources, (8) Inclusivity, Accessibility, and Navigability. 

To facilitate the result summary, the six-point Likert scale (1=Strongly Disagree, 
2=Disagree, 3=Somewhat Disagree, 4=Somewhat Agree, 5=Agree, 6=Strongly Disagree) 
is treated as an interval scale to compute means and standard deviations that describe the 


central tendency and variability of the data distribution in each of the eight categories 


(Table 5). 
Category N Mean Std Deviation 
Course Overview and Introduction (4 items) 102 19.04 1.67 
Assessment of Student Learning (4 items) 100 18.76 2.05 
Instructional Materials and Resources Utilized (3 items) 93 14.01 1.89 
Student Interaction and Community (2 items) 99 9.10 1.78 
Facilitation and Instruction (4 items) 97 15.56 230 


Technology for Teaching and Learning (2 items) 93 9.05 L533 
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Student Support and Resources (2 items) 89 =9.10 Lae 
Inclusivity, Accessibility, & Navigability (4 items) 95 18,69 2.18 
Table 5. Descriptive Statistics of the CSUB Student Survey Responses 
When the number of items is considered in the student ratings, the per-item 
responses reached a point above 4.5 on the six-point scale across all eight categories in 
Table 5. The highest average rating of 4.76 occurred in a category of Course Overview 
and Introduction while categories Student Interaction and Community and Student Support 
and Resources tied at the lowest rating of 4.55. With a rating range less than 0.11, the 
results demonstrated consistent approval of CSUB online and hybrid instruction by the 
survey respondents across the eight scales of SCSD. 
It should also be noted that the response counts (N) vary across these eight scales in 
Table 5 due to missing data in the survey outcomes. However, the rate of missing responses 
is less than 10% of the accessible sample, which is below the nonresponse tolerance 
threshold set by the standard of the National Center for Education Statistics*. Details of 
the survey findings at CSUB, including the sorted results for free-response items, can be 


found from the previous version of the CO report, “Report for the CSU Quality Assurance 


Program: Survey Findings about CSUB Student Learning Experiences in Summer, 2017”. 


Category Label X,  X, df tp _ Effect Size 


Cl LS AO oe 26 0.26 
C2 18.39 19.98 98 1.40 .16 0.28 
C3 1S Ae SO 06 29 0.22 


3 https://nces.ed.gov/statprog/2002/std4_1.asp 
4 http://www.csub.edu/~jwang/TLC.pdf 
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C4 8.61 941 97 2.24 .03 0.45 
CS 17,95: 18195.) 95; 22,085.04 0.43 
C6 8.94 912 91 0.54 .59 0.11 
C7 So. Noe Ot or 0.50 
C8 18.19 19.02 93 1.82 .07 0.38 


Table 6. Test of the Rating Differences between Online and Hybrid Student Respondents 


Among the 103 respondents, 39 students enrolled in hybrid courses and 64 students 
took online courses in AY 2016-17. To accommodate the margin of the tabulation, these 
categories are labeled sequentially as C1-C8 in Table 5. The mean ratings from the hybrid 
(X,) and online (X,) respondents are subjected to statistical testing in Table 6. Following 
the guideline from an APA publication manual (American Psychological Association, 
2001), effect sizes are computed to avoid statistical artifacts due to large samples. 

Although the survey ratings are more positive from online students than their peers 
in hybrid courses across all eight categories (Table 6), the statistical testing shows 
significant differences in the survey findings across categories of Student Interaction and 
Community (C4), Facilitation and Instruction (C5), and Student Support and Resources 
(C7) at a=.05. The corresponding effect sizes are above 0.40, confirming a moderate 
impact in student responses due to the online and hybrid course settings. 

In summary, the student survey indicated a positive rating for online and hybrid 
instruction across eight categories in Table 5. Online instruction was viewed by the 
respondents as a more supportive platform than hybrid teaching in terms of enhancing 


student interaction, community, and resource supports (Table 6). 
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Conclusions 

This report is built on collaborative efforts across several offices of CSUB to merge 
a comprehensive database for examining the impact of QA professional development on 
faculty teaching performance and student learning outcomes through online and hybrid 
instruction. The institutional support for faculty training has led to rapid growth of student 
enrollment in online and hybrid instruction. Accompanied with the participation of local 
instructors in QM and/or QOLT reviews are certifications of nearly 200 courses across all 
four academic schools at CSUB. 
In this study, the effectiveness of quality assurance is confirmed by a comparison of student 
performance data across different teaching platforms. In particular, students in online and 
hybrid classes demonstrated a higher chance to achieve an A grade than their peers in a 
face-to-face class (see Table 2). The percent of students receiving a passing grade or 
above was also higher in online and hybrid courses than face-to-face courses (see Table 
3). Across demographic classifications of the student body on UMI, Gender, Pell-Eligible, 
and Remediation dimensions, students in online or hybrid courses earned slightly higher 
GPAs than their peers in face-to-face classes (see Table 4). The positive outcomes were 
substantiated by consistent results of a student survey pertaining to the QM CORE/QOLT 
essential standards (Table 5). 

In comparison to the previous version of the CO report, the scope of this report is 
expanded beyond an analysis of the student survey data last Summer to include results of 
transcript analysis, training record aggregation, demographic information gathering, and 


probabilistic inference. As a result, the report development has completely conformed to 
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the Outline for Reporting SQuAIR Results, August 2017, a guiding document from the 
Chancellor’s Office. Based on the resulting triangulation, this report concludes with an 
assertion to support the CSU system-wide capacity building through the SQuAIR project. 
To sustain the ongoing progress, FTLC of CSUB is introducing a new program 
this academic year to guide faculty through the development of online or hybrid courses. 
Ten participants will be recruited in the Online Academy with a purpose to provide them 
with the knowledge and support they need to build a complete course and become certified 
to teach online. Participants will be given opportunities to collaborate with one another 
and utilize available facilitators to brainstorm, organize and develop their course. Upon 
completion of the monthly workshops, they will submit courses for an internal review to 


complete the certification process. 


Recommendations 

Because this first project guided by the online reporting template, it is prudent to 
identify opportunity for improvement prior to the report completion. In this regard, it 
should be noted that the SQUAIR Survey data were not gathered from a random student 
sample, nor did the student sampling cover a contrast group from face-to-face instruction 
for result comparison. As a result, the CSUB respondents came from course sections taught 
by 10 instructors in AR 2016-17. An analysis of GPA from these classes showed 
significant differences between the surveyed students and the general student body at 
CSUB in academic achievement, which justified speculation on confounding variables 


behind the population representation. 
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Like in any data matrix presentation, subject representation only represents one 
dimension of the survey design. On the variable dimension, it was already noted on page 
12 that CSUB did not have adequate data to differentiate the first-generation college 
students, nor did the university have the legal authority to require that information for 
student admission. In addition, half of the 10 instructors who taught courses for these 
SQuAIR survey participants did not provide demographic information, such as years of 
teaching online or in higher education. The CSUB team conducted a sensitivity analysis 
to assess the impact from the missing demographic variables. While the current literature 
indicated the length of teaching experiences as an indicator to differentiate novice vs. 
experienced instructors, researchers reported that “most new instructors (those with less 
than one year of teaching) were very receptive to and appreciative of” the professional 
development pertaining to online teaching°. It was also reported that some experienced 
(more seasoned) faculty members were more resistant to engage in professional 
development°. Thus, it remains unclear whether the exclusion of certain demographic 
variables will necessarily undermine validity of a project like SQuAIR. 

In summary, consideration of the future recommendations is inseparable from the 
voluntary natures of the data gathering. For instance, 64 out of the 103 CSUB participants 
in the last SQUAIR survey came from a class section that was taught by one instructor. For 
the remaining one third of the instructors, each received only one student response. In 


retrospect, the Chancellor’s Office was considerate in clarifying the original intention of 


5 http://www.westga.edu/~distance/ojdla/summer162/eskey_roehrich162.html 
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the information gathering, i.e., “This survey is NOT an evaluation of your instructor’s 
teaching performance”®. It seems important to continue the practice of not only indicating 
what the data are intended to represent, but also clarifying what data are NOT designed 


to measure. 


5 http://www.surveygizmo.com/s3/2746632/CSU-QA-Course-Survey 
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